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EVALUATING MENTAL HEALTH CONSULTATION: 
PROBLEMS IN RESEARCH APPRAISAL 



Philip A • Smith 
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Northridge, California 

Abstract 

Most programs of mental health consultation are 
offered without systematic plans for follow up eval- 
uation, Controlled experimental procedures for as- 
sessing consultation outcome are difficult to apply 
in most field settings, and alternate approaches to 
evaluation are needed* Eight problems which are com- 
monly encountered when undertaking objective study of 
the consultation process ar.e described, and the crit- 
ical incident technique is suggested as a useful post 
hoc assessment tool in situations where experimental 
controls are lacking. 



k\ 



o 

o 



ERjt 



CP 

c? 



1 



EVALUATING MENTAL HEALTH CONSULTATION: 

PROBLEMS IN RESEARCH APPRAISAL* 

Philip A, Smith 

San Fernando Valley State College 
Northridge, California 

In recent years the literature on community consult- 
ation has dealt heavily xvith emerging, innovative approaches 
to the delivery of mental health services in schools and re- 
lated community settings .2>3*^>6,7 There now is growing 
awareness of the need for better evaluation of various methods 
of health service delivery.-*- *^,9 ,12 Funding sources, profess- 
ional groups, and a more sophisticated general public have in- 
creased their demands for evaluation,-*^ and some writers sug- 
gest that community mental health can no longer be sustained 
without clear demonstration of success as measured by objective 
criteria.** - 

May-*-3, Ring and Yudin-*- -5, Osterweil-*-^ and others ■*■7*18 
have outlined some of the difficulties faced by the researcher 
in this field. Because of the complexities of undertaking 



*This paper is based on a report entitled "Evaluating School 
Consultation: Problems in Objective Appraisal,” presented as 

a part of the symposium. Problems in School Consultation , at 
the Western Psychological Association, San Francisco, California 
April 1971. 
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evaluational studies, many mental health consultants 
operate without objective feedback about their own 
performance. As a result, highly individualized styles 
of consultation tend to develop, ineffective methods of 
consultation persist, and failures of consultation may be 
blamed on the consul tee agency rather than on the consult- 
ative approach,. Funding for some consultation programs is 
also placed in. jeopardy when program effectiveness cannot 
be demonstrated. The need for evaluational studies is thus 
clear, but there are many problems which interfere with car- 
rying out objective research in the field. This paper describes 
eight problems which are commonly encountered and outlines an 
approach which circumvents many of them. The examples dis- 
cussed are drawn from consultation in school settings. 

1. The problem of the contract and definition of goals . 
Many consultants to schools and related community groups be- 
gin their work without any formal contractual arrangements 
which specify the objectives and aims of consultation. An 
exception to this is the independent school consultant ivho 
offers services for a fee. Commonly, however, mental health 
consultation is provided without fee through a community 
agency or as part of a state or federal program which under- 
writes services to schools. In some cases consultation is 
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provided by one arm or branch of the school system to 
another, again without specific fee arrangements. Con- 
tractual arrangements are implicit rather than explicit, 
general rather than specific, and diffuse rather than well 
focused, tightly stated and specific. 

If the goals are poorly defined at the outset, their 
definition is somewhat sharpened a year later. The con- 
sultant may ask himself why he has been unable to achieve 
changes he had hoped for or the school staff may wonder why 
the consultant has failed to produce some of the things they 
had expected. Then, if not before, consultant and consultee 
should meet for specification of goals and objectives which 
they can mutually support. 

2. Shifting goals . A second problem is that of chang- 
ing goals. Sven when explicitly formulated at the outset, 
the aims and processes of consultation tend to change as the 
relationship develops. A school may find itself asking very 
different things of its consultant after it has had experience 
with him for a year or two. In fact, some consultants view a 
change in their relationship and a shift of consulting aims 
as earmarks of successful consultation. Nevertheless, both 
of these problems so far mentioned — failure to state goals 
and objectives at the start of consultation, and the tendency 
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for aims and goals to shift as consultation proceeds — 
make evaluation of the consultation process a difficult 
matter. How can one name, count, and measure variables 
that aren*t well specified to begin with and which change 
as they go along? 

3* 1 Internal 1 versus 1 external [ change . A third prob- 

lem is that of finding observable behavioral measures for 
assessing change which takes place within the consultee. 

Such internal changes as improved morale, reduced anxiety, 
more appropriate self-evaluation and the like are often 
sought through consultation, rather than readily observable 
changes which are more easily measured. In this respect, 
evaluation of the mental health consultation process pre- 
sents all the problems of personality assessment or evalu- 
ation of psychotherapy. Many consultants, like many therapists, 
proceed on faith rather than on any objective evidence of their 
own effectiveness. 

4. Environmental impact . Changes which occur in the 
environment may have an impact that overrides any “experimental 
effect” of the consultant. The Los Angeles teacher strike is 
an example of an environmental variable which had massive im- 
pact on teacher morale, team work and educational achievement. 

It was difficult to judge the effectiveness of consultation in 
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the schools — or indeed the effectiveness of any program 
change which was introduced in Los Angeles city schools 
during 1970 — because of the overriding effects of the strike* 

This is a problem which social psychologists have long faced 
in their field studies of changes which occur over a period 
of time during which uncontrolled environmental variation 
may take place* 

5 ° Delayed action, triggering, or "ripple” effects . The 
effects of consultation may be cumulative, or trigger indirect 
changes which are noted considerably after consultation is 
introduced. Does one measure the initial "splash” of con- 
sultation or the ripples which move across the system over a 
period of time? For example, the growth which a consultant 
may help an administrator to achieve can lead to changing re- 
lationships between administrator and teacher, and these effects 
may in turn be felt by the students as teacher performance in 
the classroom changes. What does one measure: Changes in the 

administrator, the effects which these create on teacher at- 
titudes, changes in classroom climate which may occur, or per- 
haps ultimately the reduction of behavioral disturbances by j 

students in the school? Does one measure the immediate effects 

or the delayed ones? The problem of evaluating consultation j 

| 

effectiveness is thus compounded by the possibility that maxi- { 

mum change may occur well after the consultant has left the j 

i 

system* j 
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6. Baseline measures or the "entry” problem . If pre- 
and post- measures of change due to consultation are desired, 
initial baseline measure must be made at the time of entry, 
i.e. the outset of consultation. However, the consultant 
ordinarily does not enter a system with sufficient acceptance 
to permit him to measure such variables as teacher attitudes, 
student morale, classroom climate or administrative style, 
much as he might wish. After one or two years of consulta- 
tion the system may permit assessment of these variables. 

However, if the system was unwilling to permit baseline meas- 
ures to be taken at the start of consultation, how can change 
be assessed? It is important to find- a way of measuring change 
after-the-fact where the entry problem makes acquisition of 
baseline data impossible. 

7. Hawthorne effect . When measurable changes do occur, 
are they the result of the consultants particular approach, 

or is it simply that change occurs as a result of system aware- 
ness that someone new is present and is trying to make conditions 
better? Until we find accurate ways to characterize different 
consultation styles and procedures and until we are able to 
specify and measure changes in our end variables, we will not be 
able to answer this question. 
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8, Paradoxical change. If we are not careful in thinking 
through what it is we are measuring, we may find paradoxical 
increases in behavior base rates which at first seem to in- 
dicate unfavorable changes in a system. 11 For example, the 
number of children added to group counseling, the number of 
referrals for behavior problems, or the number of classroom 
crises which are brought to attention may be a result of better 
case finding and greater willingness by teachers to deal with 
problem behaviors which had previously been kept covert. 

In the light of these many problems, what tools do we 
have at our disposal which may be used in the evaluation pro- 
cess? I faced this question a year ago when as a member of a 
mental health consultation evaluation team I undertook the 
pilot evaluation of a fellow consultant who had provided serv- 
ices to a junior high school in the Los Angeles area. Various 
members of the team suggested, variables which should be measured 
and I constructed a short questionnaire 1 ^ to serve as an inter- 
view guide in discussions with school staff about critical in- 
cidents which had occurred during the consultation period. The 
school had had three consultants over a two-year period and no 
initial baseline measures had been taken. Xet by examining 
critical incidents in several areas — student behavior, teacher 
behavior, and administrative behavior — it became possible to 
characterize the nature of consultation which had been offered. 
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the degree of acceptance by school staff of each of the three 
consultants, and the types of changes which were occurring as 
a result of consultation. ^ 

The critical incident technique lends itself readily to 
the field of mental health consultation. Originally utilised 
to gather research data in situations where experimental studies 
could not be arranged (for example, Fitts and Jones study Of 
the causes for aircraft instrument failures and near-fatal air 
accidents ), 10 the technique calls for asking persons involved 
in critical incidents to describe the circumstances fully, re- 
count their actions, and identify any causative variables they 
can. The assumption underlying the method is that from a large 
number of incidents the investigator can discover variables 
which are critical, i.e., which have led to success or failure*^ 

Although consultees had difficulty describing incidents 
which they felt were "critical," they were rather easily able 
to describe occurrences which were "typical " 1 or illustrative" 
(both somewhat softer terms). These incidents tend to be the 
grist of the consultant's mill and offer rather sharp insight 
into his activities and how they are received by cansultee. 

The process of reviewing the effectiveness of consultation, 
and discussing critical incidents which had occurred was also 
helpful to school staff in thinking back over their pattern of 
utilizing consultation, and thinking ahead to ways of strength- 
ening their use of this resource. The value Of consultation 
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becomes more apparent when its metamorphosis over a period 
of years is viewed* It was^possible even to make rough es- 
timates of dollar savings which had been brought about* 

A thumbnail history of consultation can also be obtained 
with the help of a structured interview and discussion of 
critical incidents* When a new consultant enters a setting 
in which consultation services have been previously provided, 
it should prove helpful for him to review the approaches, suc- 
cesses, and failures of his predecessors* Latent needs and 
untapped areas of service become quickly apparent and an in- 
dication is gained of who the consultants active supporters 
will be* 

The importance of establishing the contract in consulta- 
tion is stressed in the literature^ but often overlooked in 
practice* A review of prior consultation, its goals, objectives 
and shortcomings, helps a school staff discover what additional 
variables they may wish to examine, alter, or attack, and per- 
mits them much more meaningfully to establish a contract which 
outlines the goals and objectives of consultation in specifiable 
behavioral terms* 

Summary * . 

Besearchers face a number of problems in evaluating the 
effects of mental health consultation* All the classic prob- 
lems of action research taking place in a field setting where 
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multiple events contribute to outcome characterize the 
evaluation problem. The goals of consultation tend to be 
implicit rather than explicit and behavi orally stated, or 
are diffuse and general rather than specific. Even when 
well formulated initially, goals change as a result of time 
and the consultation process. Unexpected environmental change 
such as a teacher strike may override any experimental effect 
of consultation. Consultation may have a ripple effect which 
extends beyond the actual period when consultation is pro- 
vided and which may bring about changes in the system con- 
siderably downstream. The measurements that one may wish to 
make in a system can sometimes be approached only after it ha£ 
been softened up by consultation. Finally, paradoxical change 
is sometimes noted as a result of better case finding, leading 
to the apparent results that behavior problems have increased 
rather than decreased as a result of consultation. The critic 
incident technique is suggested as a useful post-hoc evalua- 
tiohal tool. In addition to serving as an evaluative device 
it permits a refocusing of objectives by both consultant and 
eonsultee on their future contract. 
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